The EAGLE-3 Pipeline Comes Together: From 100K Hidden States to Deployed Speculation on Kimi-K2.5

Introduction

In the previous segment, the EAGLE-3 training pipeline for the Kimi-K2.5 language model had reached an inflection point. All 10 datasets had been generated via OpenRouter API at a cost of $86, producing 40,114 raw samples and 138.4 million tokens. The data was ready, the architecture was validated, and the path forward was clear: extract hidden states from the target model, train a draft model on 100K samples, and deploy it with SGLang speculative decoding. What remained was the execution — and as any ML engineer knows, the gap between a plan and a working system is where the real work happens.

Segment 30 of this opencode session spans that gap. Over approximately 250 messages, the assistant and user navigated a complete end-to-end pipeline: merging and shuffling the datasets, extracting 37,312 hidden state samples totaling 87.8 million tokens and ~4.6 TB of data, surviving a catastrophic VM crash and disk migration, debugging a Triton shared-memory compiler error on Blackwell GPUs, discovering that a single parameter in the DataLoader was the root cause of 8× suboptimal training throughput, training the draft model to 74.7% validation accuracy over 10.8 hours, and finally deploying the server with EAGLE3 speculation — ready for the benchmarking that would determine whether the entire effort had paid off.

This article synthesizes the full arc of Segment 30, drawing on both chunk articles [chunk 30.0] and [chunk 30.1] to trace the narrative from data preparation through deployment. The story is one of engineering resilience: of bugs that nearly derailed days of work, of infrastructure failures that tested the pipeline's design, and of the methodical debugging that turned each obstacle into a learning opportunity.

Part I: The Comprehensive Knowledge Dump — A Project's Memory

Segment 30 opens with one of the most remarkable artifacts in the entire conversation: message 4091, a comprehensive knowledge dump that documents every discovery, decision, bug fix, benchmark result, architectural detail, and plan for the entire project up to that point. Running over 10,000 words, this message reads less like a conversational turn and more like a project README, a design document, and a post-mortem rolled into one.

The message was written at a critical transition point. All data generation was complete, and the assistant was about to embark on a multi-day hidden state extraction pipeline. Rather than letting the accumulated knowledge remain scattered across hundreds of previous messages, the assistant synthesized it into a single, coherent document. The structure is meticulous: hardware specifications (8× RTX PRO 6000 Blackwell GPUs, no NVLink, PCIe Gen5), software versions (SGLang, vLLM, PyTorch 2.10.0, CUDA 12.8), model architecture details (Kimi-K2.5, 1T-parameter MoE with MLA and INT4 quantization), tokenizer critical findings (the <|im_end|> token having two IDs, one of which is wrong), the EAGLE vs EAGLE3 flag bug, performance benchmarks (90 tok/s baseline, 82.3 tok/s with EAGLE-3 speculation), and detailed space estimates for the upcoming extraction.

This document served as organizational memory — a durable artifact that could be consulted during the complex work ahead. It encoded not just what was known, but how that knowledge was obtained, what assumptions it rested on, and what decisions it supported. In a project spanning weeks and hundreds of messages, such documentation is not a luxury but a necessity.

Part II: The Merge and the Extraction Begins

Following the knowledge dump, the pipeline moved into data preparation. The assistant verified the merge script (merge_and_shuffle.py), deployed it, and monitored its execution. The merge process combined the response data from multiple datasets into a unified training corpus, then shuffled it to ensure the drafter would see a diverse mix of examples during training.

The merge completed successfully, producing 37,312 records with 87.8 million tokens — a dataset roughly 10× larger than the previous 10K-sample run that had achieved only 2.1 acceptance length. A cleanup operation followed, removing the 924 GB of raw response data that was no longer needed. This seemingly mundane deletion was a critical resource management decision: with only 11 TB of available disk space and a 3.5 TB extraction looming, every gigabyte mattered.

The hidden state extraction required a specially configured SGLang server with a hidden state dump patch applied to the model code. The assistant had to stop any existing SGLang instances, verify the patch was applied, configure the server for extraction mode (with --disable-cuda-graph and --disable-radix-cache to ensure clean hidden state capture), and launch it across multiple GPUs.

A critical moment came during server verification. The assistant polled the server's health endpoint and received a 200 OK response, but further investigation revealed the server was silently failing — the model weights hadn't finished loading, and the Python import path was incorrect. This false positive health check nearly derailed the pipeline. The fix required correcting the PYTHONPATH to include the patched model directory and ensuring the SGLang server was launched with the right environment.

Part III: The Counter Bug and the Race Condition

The extraction launched, but almost immediately encountered a subtle bug. The hidden state dump mechanism was using a shared counter to match dumps with requests, and this counter was being corrupted by concurrent requests. The symptom was that dump files were being associated with the wrong requests, producing corrupted training data.

The debugging process was extensive. The assistant examined the SGLang server code, traced the counter's behavior across multiple request handlers, and identified the root cause: the counter was incremented in the request handler but read in a separate callback, creating a race condition when multiple requests were in flight simultaneously. The fix was to remove the fragile counter entirely and instead match dumps to requests by comparing token counts — a more robust approach that didn't depend on shared mutable state.

After the fix was deployed, the extraction ran cleanly: zero errors in the first 60 samples, then zero errors across thousands of samples. The pipeline had achieved reliability.

This bug is a cautionary tale about shared mutable state in distributed systems. A single integer counter, incremented in one thread and read in another, nearly invalidated days of work. The fix — matching by token count rather than by counter — replaced fragile synchronization with deterministic matching. It is a lesson that applies far beyond this specific pipeline: any time you find yourself relying on shared mutable state across concurrent operations, you are one bug away from silent data corruption.## Part IV: Infrastructure Catastrophe and Recovery

The extraction was progressing smoothly when disaster struck. The underlying Ceph cluster ran out of storage space, forcing a VM kill and a complete disk migration to a new 15 TB NVMe drive attached directly to the host machine. The VM crash was catastrophic: all running processes were terminated, the SGLang server was killed, and the extraction script was lost from /tmp.

The recovery process was methodical and multi-stage. First, the assistant assessed the damage: a vLLM inference server had auto-started via a systemd service, consuming all 8 GPUs. The assistant stopped and disabled the vLLM service, reclaimed the GPU memory, and verified that the hidden state files (18,421 previously extracted samples) had survived the crash intact on the migrated disk.

The extraction script had to be re-uploaded, the SGLang server restarted, and the extraction relaunched with resume capability. The resume feature was critical — it allowed the extraction to pick up from sample 18,421 rather than starting from scratch, saving over 24 hours of work. The assistant verified the recovery by checking that the extraction was producing new files and that the log showed it was processing samples beyond the previous checkpoint.

This recovery sequence demonstrates the importance of designing systems for resilience. The extraction script's resume capability, the persistence of hidden state files on the migrated disk, and the assistant's methodical approach to verification all contributed to turning a catastrophic failure into a manageable delay. Infrastructure failures are inevitable at scale; the question is whether your pipeline is designed to survive them.

Part V: The Moment of Completion — Extraction Crosses the Finish Line

After the recovery, the extraction continued smoothly. The assistant and user entered a phase of monitoring and verification. Progress checks revealed the extraction at 34.5%, then 49.2%, then 68.6%. Each check was a moment of verification — confirming that the system was still healthy, that errors remained at zero, and that the ETA was holding steady.

The extraction completed with a final count of 37,312 samples and zero errors. The user's message "GPUs idle now" signaled the milestone — the GPUs had transitioned from active inference to idle, and the extraction script was in its final CPU-only post-processing phase. However, the transition was not seamless. The assistant assumed that building sample_lengths.json would be a quick metadata step, but discovered that it involved scanning all 37,312 .pt files — totaling 3.5 TB — and loading each one to measure its sequence length. This led to a brief detour where the assistant investigated whether the file was even needed for training, ultimately determining that the training pipeline reads .pt files directly and doesn't require sample_lengths.json. The scan was killed, and the pipeline proceeded.

Part VI: The Architecture of Frozen Knowledge — Understanding Parameter Freezing

Before training could begin, a critical architectural question needed resolution. The user asked: "Is anything frozen?" This seemingly simple question triggered a deep investigation into the EAGLE-3 training script's parameter freezing strategy.

The assistant examined the monkey-patched training script and discovered a deliberate design: embed_tokens (the embedding layer) and verifier_lm_head (the verifier's language modeling head) were frozen, while the draft model's lm_head, projection layer (fc), and decoder layer were kept trainable. This matched the upstream speculators library convention exactly. The reasoning was sound: the drafter should learn to predict tokens using the verifier's hidden states as conditioning, but the verifier's own embedding and output layers should remain fixed to prevent the drafter from drifting away from the verifier's representation space.

The user's response — "Makes sense, proceed" — was the green light that launched the 100K-scale training run. Those three words carried the weight of the entire pipeline: the data was ready, the architecture was validated, and the user had signed off on the approach.

Part VII: The First Breath of Training — Monitoring Early Convergence

With the launch command issued using torchrun --nproc_per_node=4 with a learning rate of 3e-5, cosine scheduler, 5% validation split, and 3 TTT steps, the assistant began monitoring. The first meaningful health check, taken roughly three minutes after launch, told an encouraging story.

The loss trajectory showed a total loss of ~28 across three TTT steps — "starting high" but "already dropping": loss_0 from ~9.2 to ~8.9, loss_1 from ~9.1 to ~8.2, loss_2 from ~9.0 to ~8.0 — all at a warmup learning rate of just 1.9e-6 (barely 6% of the target). This downward trajectory at a fraction of the target learning rate was a strong signal that the optimization landscape was well-behaved.

The GPU utilization probe revealed a healthy system: four GPUs running at 100% compute utilization with 28.6 GB of memory allocated and power draw of 227–297W out of 600W TDP. The remaining four GPUs sat idle in the P8 power state — exactly what one would expect from a --nproc_per_node=4 invocation on an 8-GPU node. The 100% utilization was particularly significant: it meant the GPUs were not bottlenecked by data loading, CPU preprocessing, or inter-GPU communication, validating the training configuration.

Part VIII: The 35-Hour Miscalculation — Discovering the FSDP Bottleneck

The initial training run revealed a shocking estimate: 177,230 total steps across 5 epochs, implying a training time of approximately 35 hours. This was far longer than expected for a 37,312-sample dataset. The assistant diagnosed the issue: FSDP (Fully Sharded Data Parallelism) was sharding the data across 4 GPUs, meaning each GPU only processed a quarter of the samples per epoch. Combined with the DataLoader's batch_size=1 and max_seq_len=8192, each step processed a single packed sequence — resulting in 35,446 batches per epoch.

This was a subtle interaction between FSDP's automatic data sharding and the training script's batch configuration. The assistant initially considered increasing max_seq_len to pack more tokens per batch, but this led down a rabbit hole of OOM errors and Triton compilation failures that would consume the next several hours of debugging.## Part IX: The Triton Shared-Memory Barrier — A Compiler-Level OOM on Blackwell GPUs

The quest to increase GPU utilization through larger sequence lengths hit a wall at max_seq_len=16384. The training didn't crash with a familiar CUDA out-of-memory error — it crashed with something far more subtle: a Triton shared-memory overflow.

The error message read: "RuntimeError: No valid triton configs. OutOfMemoryError: out of resource: triton_per_fused__to_copy_add_div_expand_mul_pow_sum_view_1 Required: 163912 Hardware limit:101376." This was not a GPU VRAM issue but a compiler-level failure: torch.compile with the Triton backend had generated a kernel configuration for RMSNorm that required 163,912 bytes of shared memory per block, but the SM120 architecture (Blackwell) only provides 101,376 bytes.

This distinction between HBM (VRAM) and shared memory (on-chip SRAM) is critical and often misunderstood. The assistant's diagnosis was precise: "This is a Triton autotuning issue on SM120, not a memory issue." The proposed workaround — disabling torch.compile — was pragmatic but had implications: it would break flex_attention, which is required for the block-diagonal masking in packed training. The assistant correctly chose not to pursue this path, instead stepping back to re-examine the actual data pipeline.

This debugging episode highlights a growing challenge in the ML engineering landscape. As hardware architectures evolve — Blackwell's SM120 is a new design with different resource limits than Hopper's SM90 — compiler backends like Triton must be updated to generate valid kernel configurations. Until those updates land, engineers working on cutting-edge hardware must either avoid certain operations or find creative workarounds.

Part X: The Packing Breakthrough — How Fixing Batch Size Unlocked 8× Efficiency

The breakthrough came when the assistant re-examined the collate function in the speculators library. The critical insight was that the DataLoader's batch_size parameter controls how many samples are passed to the collate function. With batch_size=1, the collate function received exactly one sample per call — no packing occurred at all. The 35,446 batches/epoch were each processing a single sample, which explained the low GPU utilization: the GPUs were spending most of their time on overhead rather than computation.

The fix was elegant: increase batch_size to 8 while keeping max_seq_len at 8192. The collate function would receive 8 samples, concatenate them (~18K tokens total), then truncate/pad to 8192 tokens. This produced tightly packed batches where multiple shorter sequences filled the attention window, with block-diagonal masking keeping them independent.

The results were dramatic:

Part XI: The Patient Wait — Training to Convergence

With the packing fix in place, the training ran for approximately 10.8 hours across 5 epochs. The assistant monitored periodically, checking GPU utilization, loss trajectories, and convergence patterns. The first validation milestone at epoch 1 showed full_acc_0 at 73.2% and an estimated acceptance length of 2.73 — already better than the previous 10K-sample drafter's 2.1.

The final validation metrics showed a model that had converged well:

| TTT Step | Loss | Full Acc | Cond Acc | |----------|------|----------|----------| | 0 | 0.920 | 74.7% | 74.7% | | 1 | 1.919 | 50.6% | 67.0% | | 2 | 2.705 | 33.4% | 64.3% | | 3 | 3.298 | 22.0% | 63.2% | | 4 | 3.772 | 14.3% | 61.9% |

The estimated acceptance length was ~2.95 tokens, up from 2.73 at epoch 1 and 2.1 with the earlier 10K-sample model. The assistant's qualitative observations — "no overfitting," "plateaued around epoch 3-4," "well-utilized" — reflected diagnostic thinking that went beyond mere metric reporting. The model had converged to its capacity limit given the data.

The 74.7% validation accuracy at TTT step 0 is particularly noteworthy. This is the accuracy of the draft model's first-token prediction — the most important metric for speculative decoding, since a high first-token acceptance rate means the drafter can generate multiple draft tokens before needing verification. At 74.7%, the drafter is correct on its first guess nearly three-quarters of the time, which translates directly to wall-clock speedup.## Part XII: The Weight Key Fix — Bridging Training and Deployment

With training complete, the deployment phase began. The first obstacle was a weight key mismatch between the training framework and the inference engine. The speculators library saved weights under the prefix layers.0.*, but SGLang expected midlayer.*. This mismatch had caused a zero-acceptance-rate bug earlier in the session and was a known integration issue.

The assistant checked for the existing fix script on the remote machine. The output was simply "MISSING" — the script, stored in /tmp, had been wiped by the VM crash and reboot. The assistant wrote a new version locally, transferred it via scp, and executed it. The script loaded 16 tensors from the safetensors file, identified 10 keys that needed renaming, and performed the transformation. This was a quintessential "last mile" problem: without this simple key rename, the entire training investment would have been worthless.

The weight key mismatch is a symptom of a broader challenge in the ML ecosystem: the gap between training frameworks and inference engines. The speculators library was designed primarily for vLLM compatibility, but the deployment target was SGLang. Each framework has its own conventions for weight naming, configuration formats, and API surfaces. Bridging these gaps requires either framework-agnostic storage formats or careful integration work — and in practice, it often falls to the engineer to write the translation layer.

Part XIII: The SGLang Argument Odyssey — Three Crashes to Correctness

Deploying the SGLang server with EAGLE3 speculation proved surprisingly treacherous. The first attempt crashed because --num-speculative-tokens was not a recognized flag. The assistant consulted the help output and discovered the correct name: --speculative-num-draft-tokens. The second attempt crashed with an AssertionError in _handle_speculative_decoding() because the validation logic required that --speculative-num-steps be explicitly set when using EAGLE3. The assistant traced through SGLang's source code to understand the three-parameter constraint.

The correct invocation required three flags working together:

Part XIV: The Moment of Truth — Server Up and Benchmark Ready

After 890 seconds of server startup (including CUDA graph capture for the speculative decoding pathway), the SGLang server was finally healthy. The assistant reported: "Server is up (~15 min startup with CUDA graph capture for speculation). Now let me write a benchmark script and run it."

This message marks the transition from building to measuring. The entire pipeline — hidden state extraction, distributed training, weight key fixing, server configuration — had been a means to this end. The benchmark script, benchmark_eagle3.py, was designed to test multiple configurations (16, 10, and 5 draft tokens) against a no-speculation baseline, providing the data needed to determine whether the drafter actually accelerated inference.

The segment concludes with the assistant producing a comprehensive status document, re-establishing shared context before proceeding to the benchmarking that would validate the entire effort. The drafter checkpoint at /data/eagle3/output_100k_sglang/4/ was prepared with a vLLM-compatible config, the weight keys were fixed, and the SGLang server was deployed with EAGLE3 speculation at 16 draft tokens. The infrastructure was ready. The rest was measurement.

Themes and Lessons

Several themes emerge from this segment that are broadly applicable to ML engineering at scale.

The value of empirical grounding. Throughout the pipeline, theoretical calculations were repeatedly corrected by real measurements. The storage estimate was validated against the 10K run's empirical ratio. The multi-GPU scaling analysis was anchored in observed single-GPU behavior. The ETA was continuously updated based on actual throughput. Theory provides a starting point, but measurement provides truth.

The fragility of distributed state. The counter bug that corrupted hidden state dumps is a cautionary tale about shared mutable state in distributed systems. A single integer counter, incremented in one thread and read in another, nearly invalidated days of work. The fix — matching by token count rather than by counter — replaced fragile synchronization with deterministic matching.

Infrastructure resilience is not optional. The Ceph crash and VM kill demonstrated that infrastructure failures are inevitable at scale. The pipeline survived because it was designed for resilience: the extraction script supported resume, the hidden state files were persisted on stable storage, and the recovery process was methodical rather than panicked.

The data pipeline is the bottleneck. The single most impactful optimization was not a model architecture change or a hyperparameter tweak, but fixing the DataLoader's batch_size to enable proper sequence packing. This 8× reduction in batches transformed a 35-hour training run into a 10.8-hour one. When GPU utilization is low, look first at the data pipeline.

Compiler-level failures require hardware-level understanding. The Triton shared-memory OOM on Blackwell GPUs could only be diagnosed by understanding the distinction between HBM and shared memory, and the specific constraints of the SM120 architecture. As hardware architectures evolve, engineers must develop a deeper understanding of the compiler and runtime stack.

The last mile is the hardest. Weight key mismatches, argument name typos, and assertion failures consumed more debugging time than the training itself. The gap between training and deployment is where the most brittle, framework-specific work happens — and it deserves as much engineering attention as the core ML pipeline.

Documentation as organizational memory. The comprehensive knowledge dump at the start of the segment was not a one-time artifact. It was referenced throughout the pipeline, updated as new discoveries were made, and served as the shared context that enabled efficient communication between user and assistant.

Conclusion

Segment 30 of this opencode session captures the complete lifecycle of a complex ML engineering project: from knowledge synthesis and data preparation, through extraction and debugging, past infrastructure catastrophe and recovery, to training and deployment. The final result — a drafter with 74.7% validation accuracy and an estimated acceptance length of ~2.95 tokens — represents a significant improvement over the previous iteration. But the real achievement is the pipeline itself: a system that survived bugs, crashes, and uncertainty to deliver a working model.

The EAGLE-3 drafter that emerged from this pipeline — trained on 37,312 samples across 5 epochs, achieving 74.7% validation accuracy — represents not just a trained model, but a validated end-to-end system for speculative decoding on the Kimi-K2.5 architecture. The server was deployed and ready for benchmarking. The rest was measurement.

The lessons from this segment extend beyond EAGLE-3 training. They are lessons about how to build robust ML infrastructure: ground your estimates in empirical data, design for failure, document your knowledge, and maintain a tight feedback loop between monitoring and action. In the long march from data to deployment, these practices are what separate projects that succeed from those that collapse under their own complexity.